Welcome! 


To the 3" Community Over Code Performance Engineering Track 


Track chairs: Paul Brebner, Roger Abelenda 


1st New Orleans 2022 


Technologies: Lucene, Ozone, Kafka, Cassandra, JMeter, and Spark/ML 
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2nd Beijing 2023 


Technologies: Kafka, JMeter, Arrow, Java profiling, Spark 8 Flink, Hadoop 


3rd Halifax 2023 


Technologies: Kafka, Ozone, Cassandra, Camel, JMeter/Selenium, Lucene 


Talks (3 x 2 = 6) 


11:20 Paul Brebner 
Developing Fast Applications With Open Source Software - Without The Fury (Kafka) 


12:10 Duong Nguyen, Tanvi Penumudy, Ritesh Shukla 
Design patterns and then the road to realize billions of objects, and exabytes of capacity, while preserving performance 
in Apache Ozone 


--- LUNCH 


2:20 German Eichberger, Pallavi lyengar 
Performance measurement and tuning of Cassandra 5.0 transactions on Cloud infrastructure 


3:10 Otavio Piske 
Hunting Performance Monsters on the Back of a Camel 


--- COFFEE 


4:10 Roger Abelenda 
Quick load testing from Selenium scripts 


5:00 Stefan Vodita 
Lessons Learned from Benchmarking Amazon's E-commerce Search Engine 
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Fast Open Source Software — 
Without The Fury = 


Paul Brebner - Open Source Technology Evangelist 
www.instaclustr.com/paul-brebner/ 


Community Over Code Halifax October 2023 
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Massive amounts of datais 
captured and used to gain 
milliseconds of performance 
improvements on race-days 


Using NetApp Cloud and Storage 
Technologies 


“Fast & Furious” Cars for my Grandson 


(unopened as they are “collectables”) 


(Source: ) 


(Source: wikimedia.org) 


Cars run on roads 
S/W runs on H/W 


(Source: nedia.org) 


Too Many Kafka Topics 
Slow Consumers 
Too Many Kafka Partitions 


Single Threaded Consumers— 
concurrency limited by 
partitions 


(Source: Shutterstock) 


Operational Problems 


(Source: Shutterstock) 


Cloud Platform for Big Data 
Open Source Technologies 


Focus of this talk is on 
Apache Kafka”? 


Streaming Customer Analytics 


Gaming Social 


STORE STREAM ANALYZE SEARCH ORCHESTRATE 
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Ops Procedures Prod-Ready 
TECHNOLOGY Expert Support and Automation Architectures 
Provisioning Scaling Backup and Restore 
PLATFORM Monitoring Security Service Operations 
Functional Integrations 
Application Continuous Multi-Region and 
Console Maintenance Multi-Cloud Replication 


PCI-DSS and SOC 2 Security Certifications 


24x7 Expert Support 


Kafka is a distributed streams processing 
system—it allows distributed producers to send 
messages to distributed consumers via a Kafka 
cluster. 


Producer Producer Producer 


Kafka Cluster 
Topic Topic 
Message flow 


Consumer Consumer Consumer 
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Kafka write scalability: 
Multiple messages can be written concurrently 
Using multiple partitions on different brokers. 


Kafka cluster 
6 brokers replication 3 
1 topic 4 partitions 


Send messages to topic 


11 | M2 | Topicis replicated over 
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ipie partitions and brokers 


Concurrent writes 
To leaders on brokers 1 and 6 
and followers on brokers 2, 3, 4, 5 


Partition 1 
Leader 


N 
Partition 1 
Follower 


M2 concurrently sent to leader 
of partition 2 on another broker 


Partition 3 Partition 3 Partition 3 
Follower Leader Follower 


Kafka read scalability: 


| Consumers : 
(Consumers Multiple messages can be read concurrently 


Read messages from topic From leaders of multiple partitions and brokers (and followers) 


Kafka cluster 
6 brokers replication 3 
1 topic 4 partitions 


Concurrent reads 
From Brokers 1, 2, 5, 6 


Partition 1 
M Follower 
Bkoker 6 

Partition 2 Partition 2 
Follower Leader 
Partition 3 Partition 3 
Follower Follower 
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Partition 2 
CZ ZZZ 


Partition n 


Partitions enable Consumers to share work 
(c.f. Amish Barn raising) within a consumer group 


Consumer Group 


A Consumer 


AA Consumer 


AA Consumer 


Consumers share 
work within groups 
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Multiple Groups Enable Message Broadcasting 


Consumer Group 


Partition 1 


Consumer Group 


Messages are 
duplicated across O 
Consumer groups Consumer 


Messages are duplicated (c.f. clones) across groups, — 


as each consumer group receives a copy of each message — 
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Too Many Kafka Topics 


Goods Warehouses Trucks 


Boundary 
Sensors, RFID 


The Virtual World 


kibana 
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1. Many (100s) of Topics 
100s of locations (Warehouses, Trucks) 


Each location has a topic and multiple 
consumer groups (so all the Goods in a 
location receive relevant events) 


Many topics 


Many consumer groups per topic -> 
high fan-out 


Kafka: many topics, 1 per location, Goods 
Locations many consumers, 1 per goods Kafka Consumers 


Location 1 Topic 


Location 2 Topic Consumer. 


Location 3 Topic 


Location 4 Topic 


One topic for all locations 


Using an external notification 


mechanism for event broadcasting 


(Guava Event Bus) 


Locations 


Kafka: one Kafka sensor topic, one consumer, 
many EventBus location topics 


EventBus 
location 
1 topic 


EventBus 
location 
2 topic 


Sensor Topic 


EventBus 
location 
3 topic 


EventBus 
location 
4 topic 


Goods 


Single topic, single 
1200000 consumer group, 
1120000 


155 times 
better! 


Many topics, many 
consumer groups, 7200 


Many topics Single topic 
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Train in the Canadian Rockies (Source: Getty Images) 


(Source: Shutterstock) 


(Source: Shutterstock) 


Vertical/Scale-up 
Increase Node 
Sizes 


(Source: Shutterstock) 


Fury 2: Slow Consumers 
Kafka+Cassandra Anomaly Detector 
Application 
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Kubernetes Son Nodes 
(VP 


AWS Region 


kubernetes 
IP Discovery 


Instaclustr 
Provisioning 
KA API 
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Instaclustr 
Managed Instaclustr 


Instaclustr Services Cassandra Cluster 
(VPC3) 


Kafka Cluster (VPC2) 


Cassandra 
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Initially just increased h/w resources 
Scaling was “easy” with Kubernetes 
Easy to create lots of consumers (100s) 


Initially single threaded Kafka consumer 
(no thread pools) 


Anomaly Detection Application Design 


Anomaly detection pipeline 


Kafka Consumer 


(thread pool 1) 
Get 


transaction 
to check 


u___ 
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Kafka Write 


Cluster transaction 
to C* Query 


| Cluster Size Get historic 


data 
Cassandra Client 
(thread pooll2) 
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Run fraud 
detector 


Data write path 


Transaction 
Generator 
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Cassandra 
Cluster 
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But Scalability Not Great: oe 
Scaling Is Too Easy—Scalability Harder =" 


Total Cores vs. Billions of Checks/Day (pre-tuning) 


Default Kafka consumers: 
Are single-threaded 


If the processing is “slow” then gueuing 
occurs—as the thread is blocked—reducing 
throughput 


Solutions include speed up the processing, or 
increase the number of consumers 


But more consumers > more partitions 


As each consumer needs 1 or more partitions 
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(Source: Getty Images) 


Anomaly Detection Application Design 


Anomaly detection pipeline 


Kafka Consumer 


(thread pool 1) 
Get 


transaction 
to check 


nen 


Ý 


Write 
transaction 
to C* 


data 
Cassandra Client 
(thread pooli2) 


Run fraud 
detector 


Data write path 


Transaction Kafka 
Generator Cluster 


ata read path 


data for detector 


Guery Cassandra 
Cluster 


Result? 
QO % 


Slow consumers > 
need more consumers 
and also more 
partitions for higher 
throughput 


But more consumers 
is slow, try speeding 
them up 


(Source: Getty Images) 


The famous Bondi Ocean Pool 
(in Sydney, Australia) has 2 pools 


(Source: Shutterstock) 


Anomaly Detection Application Design 


Anomaly detection pipeline 


Less consumers 
(around 100) gives 
higher throughput— 
a surprise! 


Kafka Consumer 
(thread pool 1) 


Data write path 


Cassandra 


Cassandra Clle 
(thread pool 


data for detector 


Speed up polling (thread pool 1) 
Maximize anomaly detector 
concurrency (thread pool 2) 


—Reduces the number of 
consumers and therefore partitions 
needed and gives higher 
throughput— why? 


Don’t more partitions give higher 
throughput?! Answer in part 3. 


A A :nstaclustr 
Scalability Post-Tuning—7.5 to 19 Billion 


Checks/Day—2.5 Times Improvement 


Total Cores vs. Billions of Checks/Day (pre-tuning) 


Billions of checks/day (pre-tuning) — Billions of checks/day (post-tuning) 


Fury 3: Too Many Partitions 
What's really going on under 
the Kafka Bonnet? 


(Source: Adobe Stock) rt 
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Partitions = Pistons (Cylinders) 


(Source: Getty Images) 


But How Many? 


Isetta (bubble car) 
single-cylinder, 10HP, 
top speed 55MPH 


Isetta 1-cylinder car 


...16 Pistons Is a Lot! 


a 
| 


175HP, 
100 MPH! 


Can You Have “Too Many”? 


Experimental 42-cylinder 
2,350 hp (plane) engine! 


Benchmarking (2020): Partitions and str 


Replication Factor Are the Culprits »]»]»]»]»]”]” oom 


Kafka Partitions vs. Throughput 
Cluster: 3 nodes x 4 cores = 12 cores total 


iai Replication Factor 3 (TPS) ——Replication Factor 1 (TPS 
You need sufficient eplication Factor 3 (TPS) eplication Factor 1 (TPS) 


partitions to benefit 
from the cluster er 


concurrency— 
And not too many 


that the replication 
overhead impacts 
overall throughput 


100 


Partitions 


2022 Kraft/Zookeeper Modes vs. neu 
2020 Results Better, 1000 Partitions Is Ok = 7% 


Partitions vs. Throughput (M TPS) 


ZK TPS (M) ——-KRAFTTPS (M) ——2020TPS (M 


2022 - Better 


—_ 


2020 - Worse 


10 100 1000 100 


Fury 4: Single Threaded Consumers 
What's New? Kafka Parallel 
Consumer = A New Engine! 


"La Jamais Contente", first car to reach 100 km/h Rimac Nevera — Electric “Hypercar” 
in 1899 — 68hp electric! 4 Engines, 1,888 HP, 0-400KM/H in 29s 
jep LISI) (Source: Wikimedia) 
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Theory: Little’s Law: 
Concurrency = Throughput x Time = 


Throughput = Partitions/Time 


10000000 


TT Rearranged: 
1000000 PE 


/ e Throughput = 
100000 / Concurrency/Time 


Concurrency is Partitions = 
Consumers 


Using default consumer the 
throughput drops with 
increasing time 


Only solution is to increase 
400 600 1000 1200 partitions 


Partitions=Consumers 
Maximum Throughtput (time = 1ms) Maximum Throughtput (time = 10ms) Maximum Throughtput (time = 100ms) 


Maximum Throughtput (0 processing time, but 10,000 TPS per partition limit) 


For Given Target Throughput (1M TPS) instaclustr 
Increasing Partitions With Increasing Time Des 


Partitions required for 1M TPS Target 
with increasing processing time 
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Partitions 
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(Source: Adobe Stock) 


Multiple ordering options—c.f. default Kafka only guarantees order within partitions! 


Concurrency from 1 to lots —depends on client resources, and Partitions/Key space sizes 
KEY has higher concurrency than Partition and is ordered by KEY—reasonable compromise 


UNORDERED is unordered 


(Source: Getty Images) 


10000000 

1000000 

100000 

10000 

1,000 partitions ME 
100 consumers max 100 
10 
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Throughput Records/s (1000 partitions) 


m (1) Default Consumer, RT=10ms 

2) Parallel Consumer, RT=10ms, Partition Order 

3)Parallel Consumer, RT=10ms, Key Order (1,000 keys) 

4) Parallel Consumer, RT=10ms, Key Order (10,000 keys) 
m (5) Parallel Consumer, RT=10ms, Key Order (100,000 keys) 
) 


6) Parallel Consumer, RT=10ms, Unordered 
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| instaclust 
Experimental Results: nstaclustr 


3, 50, and 200 times improvement, unordered best = 


Experimental Throughput Results, RT = 10ms 
1 Consumer, 10 Partitions, 100 Keys 
100000 


17465 20000 


1 consumer 


10 partitions 


100 keys 


10ms latency 


Partition, 1 thread Partition, 10 threads Key, 100 threads Unordered, 200 threads 
Measured TPS Theory TPS 


Too many topics 


(= too many partitions) E Too many consumer groups 


Insufficient/ 


too many partitions 


Slow consumers M a 
(= too many partitions) Au Single threaded 
tl im, consumer 


(= too many partitions) 


Minimize Topics and Partitions = Tracks 
- Buses are fast 8 self-driving on tracks 


Minimize Consumer Groups = Interchanges 
- At interchanges, buses fan out onto roads, 
reducing passenger transfers 


Maximize Consumer Concurrency = Buses 
- Multiple passengers, 
integrated with road system 


Adelaide’s O-Bahn Busway train-bus system 


ta —— 
Pit Stop Performance Penalties (Source: Getty Images) ur 
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Kafka Cluster CPU Utilization (0-100%) 


Abnormal Rapid increase in CPU Utilization 


from normal of 50% to lots 


What's going on? 


This was the only example of a 
Kafka performance problem in 
our Post-Incident Reviews 


Has the workload increased? 
No. 

Has the cluster capacity 
decreased? (e.g. lost some 


brokers) 


No. 


Attempt 1: Replace 1 broker at a time 


* Didn't work — problem reappeared when new broker took over 


partition leadership 
e (A broker restart is what triggered the problem) 


Attempt 2: Stop all customer clients 
(producers/consumers) 
e Perform a rolling restart of Kafka cluster 
e Restart clients, hold breathe... 


Apollo 3 stage J-2 Engines (13,000,000 HP) were designed 
to restart — but failed to restart in uncrewed Apollo 6 
(Source: NASA) 


SATURN V 


LIQUID HYDROGEN 
mene 3D INSULATION 
LAS VENI i AUXILIARY PROPULSION 
ANS STEM MODULE 


AFT SKIRT 


FORWARD S w 
SKIRT ~_ & HELIUM SPHERES 
3 n AFT INTERSTAGE 
| FUEL LEVEL > EMI pa 
N TL SII 
< ) = > j) 


i SENSORS 


COLD HELIUM 
SPHERES 


COMMON 


LIQUID OXYGEN TANK 
ULLAGE MOTORS(2) 
SEPARATION 
NAA PLANE 
rd THRUST 
STRUCTURE 
SATURN V 


mstaclustr 
Back to normal 


Kafka Cluster CPU Utilization (%) 


| Producer | | Producer Producer 
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Kafka Cluster 


Topic Topic Topic 


Partition Partition Partition | Message flow 


Partition [Partition Partition | 
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Network Processor Idle (%) MSI 


Kafka Network Processor Threads handle client network data 
Network Processor Idle % has decreased (more is better, less is worse) 


The client load had increased. 


Kernel Error 
e TCP: reguest sock TCP: Possible SYN Flooding ... 


Decreased Network Processor Idle % was a symptom 
e Of repeated Kafka producer connection attempts 


e TCP congestion control and window size dropped to very small and 
inconsistent values between producers and broker 


* Making it impossible to reconnect producers to brokers after a broker restart 
Cause? 
* Broker restart triggered the problem 


e Permanent fix required Linux Kernel and Kafka version upgrades 
* And different settings for SYN cookie options 
* Probably related to KAFKA issues 9648 and 764 


Aston Martin DB5 


“Ejector seat? You're joking.” 
My now “collectable” but played with Corgi DB5 (Source: Paul Brebner) 
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